Parallel Rule Mining with Dynamic Data Distribution under Heterogeneous Cluster Environment

Big data mining methods supports knowledge discovery on high scalable, high volume and high velocity data elements. The cloud computing environment provides computational and storage resources for the big data mining process. Hadoop is a widely used parallel and distributed computing platform for big data analysis and manages the homogeneous and heterogeneous computing models. The MapReduce framework is applied to divide and process the data and tasks as small elements. The frequent item set mining methods are applied to fetch frequent patterns from the database transactions. The parallel frequent mining techniques divide and process the data set with equal intervals. The Data Partitioning in Frequent Itemset Mining on Hadoop Clusters (FiDoop-DP) is adapted to perform the load balanced rule mining process. The Voronoi diagram based data partitioning scheme uses the transaction relationships. The partitioning process controls the redundant transactions with similarity metric and Locality Sensitive Hashing (LSH) technique. The Parallel Frequent Pattern Growth algorithm is employed to discover the frequent item sets. The parallel rule mining process is build to support dynamic data partitioning and distribution over the heterogeneous Hadoop clusters. The heterogeneous Hadoop clusters are formed with different resource level in each computational node. The data aware partitioning process is carried out with load balancing mechanism. The computational resource level is also used for the data partitioning process. The FiDoop-DP scheme is upgraded to handle the data placement with load balance under the Hadoop Distributed File System (HDFS) in heterogeneous nodes. The parallel frequent Item set mining process is improved with energy efficiency features.